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ABSTRACT 



This paper reports on a study undertaken to examine student 
usage of course webs (i.e., course materials and resources on the World Wide 
Web) in classroom situations in which computers are readily available and 
students are comfortable using hypertext systems. The author prepared course 
webs for two computer science courses at Grinnell College (Iowa) . Each course 
web contained: links to the remainder of the Web and appropriate external 
resources; standard handouts, such as syllabi, rules/regulations, and 
assignments; outlines for each class period; a short news document providing 
updates to the Web; online quizzes; and appropriate reference materials, 
including readings and external links. The courses were taught in a 
computer- equipped classroom, and students were allowed to use the computers 
as they deemed fit during class periods. Data were gathered using two student 
surveys on Web usage and Web server log analysis. A significant but not 
uniform use of Web-based course supplements during class periods was evident. 
In addition, students reported benefits from their self -directed in-class use 
of the online materials. Three tables present data on students' self-reported 
use of class outlines, overall Web usage in the first 7 weeks, and page 
access by category in the first seven weeks of each course. Contains 13 
references. (DLS) 
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Abstract: As increasing numbers of course webs are built, it is increasingly important to 
evaluate the effects of those webs on student learning. Web usage may depend on student 
expertise, classroom setting, and available computing resources. In past studies [Rebelsky 
1996b] [Rebelsky 1996a] I examined student reactions to a wide variety of web resources, 
from assignments to transcriptions of individual class sessions. In this present study, I 
examine student usage of course webs in classroom situations in which computers are readily 
available and students are comfortable using hypertext systems. For these students, there is 
significant, but not uniform, use of web-based course supplements during class periods. More 
importantly, these students report benefits from their self-directed in-class use of those online 
materials. 



1. Introduction 

The World-Wide Web [Berners-Lee et al. 1994] continues to change the ways in which we teach and are 
expected to teach. Many instructors are putting part or all of their courses on the web (for distance learning or 
as course supplements), and students are expecting to find course resources on the web. In spite of this growth, 
there is little formal or informal evaluation of usage or impact of these course webs. For example, of the 182 
full papers presented at the 1997 World Conference on Educational Multimedia, Hypermedia, and 
Telecommunications [Muldner & Reeves 1997], only six provided any form of evaluation of student usage of 
course webs, and many of those included only rough notes on observations. (However, a number of others 
evaluated student use of particular hypertext learning tools.) This lack of evaluation is surprising, given that 
"most researchers agree that insufficient evidence exists to determine whether outcomes match the proclaimed 
promises of hypermedia" [Fitzgerald & Semrau 1997]. 

As developers of course webs, we must ask ourselves how students use, react to, benefit from, and are otherwise 
affected by our course webs. The questions developers of course webs might ask themselves and their students 
include: (1) When do students access the materials? Before, during, or after each class period? (2) Which 
materials do students regularly use? Outlines? Syllabus? Assignments? Answer Keys? (3) Where do students 
access the materials? In the classroom? In their dorm rooms? In public laboratories? (4) In what form do 
students use the materials? Printed? Electronic? Both? Neither? 

In previous studies [Rebelsky 1996a] [Rebelsky 1996b] I reported on student usage of resource-intensive course 
webs that included outlines of the topics to be covered in class period, copies of anything written on the 
blackboards during class periods, assignments, selected questions and answers, traditional course handouts, and, 
in one course, transcriptions of each class period. Students reacted positively towards the online availability of 
the resources, and indicated that they appreciated effort required to produce these webs. At the same time, they 
felt overwhelmed by the materials, with many deciding that they needed to print and read everything. In an in- 
depth analysis of the effects of web usage on computing anxiety [King et al. 1997], King, Henderson, and Putt 
reported that use of course webs does not appreciably affect computing anxiety (either positively or negatively). 

The present study approaches the problem from a different perspective: If students have ready access to 
computers and the web during class time, how does that affect their perception and usage of the web? In 
particular, what types of pages do they use most frequently, when do they use them, and what do they say about 
their usage? The study is based on two courses: a second course in computer science, and a mid-level course in 
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2. Background 

In the fall of 1997, 1 prepared extensive course webs for two courses at Grinnell College, Computer Science 152 
and Computer Science 223. The design of these webs was based on previous research on student use and 
preferences [Rebelsky 1996a]. Each web contained (1) a front door with links to the remainder of the web and 
appropriate external resources; (2) standard handouts, such as syllabus, rules and regulations, and assignments; 
(3) moderate-sized outlines for each class period, developed on or before the day of each class; (4) a short news 
document documenting updates to the web; (5) online quizzes; and (6) appropriate reference materials, 
including readings and external links. The class outlines had three parts: (1) a short summary of the topics to be 
covered; (2) a list of administrative information with links to appropriate pages; (3) my notes on the day's 
topics. A typical outline is four pages long. Students were told about the course web the first day of the course, 
and were rarely, if ever, given paper handouts. These course webs can be found at 
www.math.grin.edu/~rebelsky/Courses/CS223/97F and www.math.grin.edu/~rebelsky/Courses/CS 152/97F. 

Grinnell College's Computer Science 152 is a standard second course in Computer Science, with emphasis on 
data structures and algorithm analysis. CS 152 meets four days per week. We teach the course in Java so that 
we may emphasize both object-oriented and imperative design. Grinnell has recently adopted a two 
language/three paradigm strategy in keeping with the new standards for introductory sequences [Walker & 
Schneider 1996]. In the fall term, there were seven students enrolled in CS 152, three of whom had taken a 
previous course based on functional programming, and four of whom had other exposure to programming. All 
students were familiar with the World-Wide Web and were unfamiliar with Java. The text for the course, a 
preprint of Bailey’s Java Structures [Bailey 1997] included an online software library and reference manuals for 
the library that were included with the course web. 

Grinnell College’s Computer Science 223 is a mid-level (sophomore to senior) course on issues in software 
design. CS223 meets three days per week and covers a wide variety of topics, from forming a programming 
team and building large projects to issues in user interface design. As such, it draws upon a wide variety of 
texts, from general treatises on software design [Winograd 1996] to collections of articles on programming 
[Bentley 1986] [Bentley 1988]. This makes students more inclined to rely on instructor-generated resources, 
such as the course web. Like CS152, CS223 uses Java as the main language so as to better illustrate object- 
oriented design. In the fall term, there were twenty-four students enrolled. Of these, nineteen were majors and 
five were nonmajors. Students were generally unfamiliar with Java, but were uniformly familiar with the 
World-Wide Web. About half were able to write HTML. 

Both courses were taught in the department's MathLAN classroom. This room includes sixteen HP 712/60 Unix 
workstations. In CS223, students were forced to pair up on workstations, but they were accustomed to this 
pairing from experiences in other courses. There were some in-class exercises given in both courses, as well as 
some electronic quizzes (in CS223, students could collaborate on a quiz). Students were permitted to use the 
computers as they deemed fit during class periods. The MathLAN classroom is adjacent to the MathLAN 
laboratory, where students often look at course pages before and after each class period. The MathLAN 
laboratory contains an additional sixteen HP 712/60 Unix workstations and is used as an open laboratory for 
student use and as a classroom for selected classes. 

During the term, I observed a number of students loading the course web pages during class periods, doing 
"research" on in-class questions by looking elsewhere on the web, and using the computer as a notebook. To 
ground these observations, I gathered data on how they were using computers and the web during and outside of 
class periods. 



3. Methodology 

Data on student web usage were gathered in two ways: students took two surveys on their web usage, and web 
server logs were analyzed using custom tools. The survey focused on students perceived use of the web, 
particularly of the class outlines. The logs permitted more general analysis. 

Before mid-semester break, students were given an electronic survey on the course webs and were told that the 
intent of the survey was to determine if and how the class web should be changed for the remainder of the term. 
In CS152, all seven students answered the survey. In CS223, twenty-one of the twenty-four students answered 
the survey. The survey asked students (1) how often they referred to the class outlines during, before, and after 
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class sessions (using a scale of 1 to 5 with 1 representing “never” and 5 representing “always”); (2) whether the 
outlines were a help or a hindrance or both; (3) whether they printed the outlines; (4) whether they took notes 
on the computer; (5) a number of free-form questions about the course web; and (6) some questions not 
pertinent to the study at hand. 

Because of a problem with the tool used to build the surveys, not all responses to the first survey were recorded 
accurately. The tool was repaired and the students were again surveyed about web usage at the end of the term 
as part of an end-of semester course evaluation. For this survey, all seven students in CS152 answered the 
survey and twenty-three of twenty-four in CS223 answered the survey. In the final survey, the numeric 
rankings were described only as "1 -never and 5-always." The numeric results in this paper are based only on 
the second survey. The textual responses are taken from both surveys. 

The access logs generated by our httpd daemon were analyzed with a custom Perl script. Cumulative data for 
the first seven weeks of the course were used for overall analyses. For each course, this script selected pages 
according to appropriate criteria. For accuracy, the pages counted were limited to those that were (1) part of the 
course web (i.e., contain the site root as part of their path); (2) accessed from inside the college (i.e., contain 
grin.edu as part of the requesting machine); (3) accessed from machines the instructor does not regularly use 
(the instructor's machines are never used by students, and the instructor does not view the web from student 
machines); and (4) valid requests (i.e., requests for which a page exists). 

For counts of usage of the current day’s outline "during class", time of access was restricted to the 15 minutes 
before and 10 minutes after the class period (if a class that met from 1:15 to 2:05, the times were 1:00 to 2:15). 



4. Results and Discussion 

Because there are few studies on student use of course webs, this study focused on student reactions and the 
types of pages they used. Most students report making at least occasional use of the course outlines, with the 
majority (five of seven in CS152, twenty of twenty-three in CS223) reporting that they always or almost always 
used the outlines at some point (before, during, or after each class period). Many students seem to pick a 
particular time to use the outlines, and stick to that time. Some students appear to use the outlines before and 
during class as a guide to what will happen and others use them later for review. 

Student comments described a wide variety of uses of the webs during and near class time. As one might hope, 
some used them to prepare themselves for the topics in a given class. 

I read the outlines before class so I know what to expect and to be in a proper mindset. [CS223 student] 

Before class, it’s a great way to get in the mindset of what’s going on in class that day. [CS152 student] 

Others used them as aids to recall, to answer questions they had during class, or to find other perspectives. 

Sometimes they give enough information beyond what is getting written on the whiteboard. That way, 

I can piece it together for myself without having to slow down the rest of the lecture for a silly 
question. [CS223 student] 

I really like being able to jump back to any point in the lecture if I find that there is something I didn’t 
quite catch and need repeated. [CS223 student] 

One danger of online resources like these is that they adversely affect student note-taking. Since note-taking 
has a clear role in learning [Carrier 1983], it is worrisome that some students use class notes as a replacement 
for their own. I attempt to combat this problem, in part, by using my outlines as only a rough guide for what I 
teach each day. I often develop new or modified examples, and rarely look at the outlines during class. 
Students reported that they perceived positive effects on their note-taking. For some, this is because they have 
not yet learned to take good notes: 

I have only mediocre note-taking skills so the outlines that you provide are superior to any notes I 
would take myself. [CS152 student] 

It’s nice to have a record of what happened in class (since I'm a poor note taker). [CS223 student] 
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After class, it's really beneficial to be able to compare your notes to the class notes and see if you got 
distracted/whatever or if you didn't emphasize the same things as the outlines did. [CS152 student] 

It is, however, worrisome that these students do not yet understand the implicit benefits of notetaking. 

More importantly, many reported that the in-class availability of outlines permitted them to concentrate more on 
understanding than on “mindlessly” copying what I write or say. 

Since I do not have to take many notes I am able to concentrate more on understanding what you say 
instead of writing down. [CS152 student] 

Instead of taking the time to write down everything and missing stuff, I can watch and listen more 
closely. [CS 152 student] 

I can spend more time listening to what you and other people say than take notes. [CS223 student] 

Some of those who were confident in their note-taking abilities reported that they used the outlines to 
supplement their note-taking. 

I like to organize notes when I take them and if I know what general subjects you cover, I'm happier. 
[CS223 student] 

One of the perceived benefits of hypertext is that it supports a variety of learning styles. Student answers 
suggested that this is, indeed, the case. 

I have much better comprehension of things that I read instead of hear. Since you do not follow the 
outline exactly, reading it can be a way of getting things rephrased or put differently. [CS 152 student] 

Seeing something on the screen, however, makes me much more likely to remember something. 
[CS223 student] 

It has been reported (e.g., in [Rebelsky 1996a] [Windley 1994]) that students tend to print “everything 
available” in their course webs Surprisingly, most students n these two courses reported that they did not 
regularly print the class outlines. In CS152, only one of seven students reported printing the class outlines. In 
CS223, only three of twenty-three students reported that they printed the class outlines. It is not clear whether 
or not this is because of the in-class availability of the resources, but it seems more likely that as computers 
become more available, students feel less needed for printed copies. 

A summary of student self-reports of usage of course outlines appears in [Tab. 1]. The first part of the table 
summarizes their reported usage before, during, and after class. The second part of the table summarizes the 
computed maximum and minimum for each student across the three categories. That is, if a student reports a 
value of 1 for before, 3 for during, and 5 for after, that student's maximum and minimum will be recorded as 5 
and 1. These categories are used to accommodate different strategies for using the course web. 



Course When 1 2 3 4 5 Median Mean 



CS152 Before 
During 
After 

Max/student 

Min/student 


2 


2 


1 


0 


2 


2 


2.71 


0 


2 


1 


3 


1 


4 


3.42 


0 


2 


1 


3 


1 


4 


3.42 


0 


1 


1 


1 


4 




4.14 


2 


3 


• 1 


1 


0 




2.14 


CS223 Before 
During 
After 

Max/student 

Min/student 


I 


0 


8 


5 


9 


4 


3.91 


1 


4 


2 


6 


10 


4 


3.86 


0 


2 


13 


8 


0 


3 


3.26 


0 


0 


3 


10 


10 




4.30 


2 


5 


12 


4 


0 




2.78 



Table 1: Student's self-reported use of class outlines. A rating of 1 indicates that the student 
reports never uses the resource during that time. A rating of 5 indicates that the student 
reports always using the resource during that time. The maximum and minimum categories 
reflect the greatest and least values students reported across each of the time periods. 



Clearly, different students find different times to use the web and place different emphases on web use. In 
CS152 there was one student who reported making very little use of the course web (a maximum across all three 
categories of 2). On the final survey, the student indicated a desire to use the web more but did not explain the 
limited use. In CS223 there were four students who made significant use of the course web before, during, and 
after class (a minimum of 4 across all three categories). 

One unexpected result was that self-reported usage of course webs in CS223 seems to decrease slightly during 
class. It may be that many students quickly refer to the outline at the beginning of class to get a better 
perspective on what will be covered and then do not refer back to them during class. This is clearly an area that 
should be examined further. 

Analysis of web log files shows that reported in-class usage of outlines is fairly close to actual usage. In CS152, 
the outline of the current class was accessed during class 5.9 times on average. These accesses came from an 
average of 5.0 machines. This is in keeping with the reported numbers. In CS223, the current class outline was 
accessed during class 15.0 times on average, from an average of 12.5 machines. A small number of in-class (or, 
presumably, shortly pre- and post-class) access came from machines outside the classroom, but this number was 
less than one document per day. 

Data from the web server log files was used to generate the reports on overall student web usage in [Tab. 2]. A 
short analysis of the files accessed suggested that certain sets of documents biased the results. In CS152, 
documentation that accompanied the book was included in the counts, so a separate analysis without that 
documentation was done. In CS223, some documents generated by javadoc included images that were counted 
towards the totals. In addition, a directory loop in the examples section resulted in their being multiple paths to 
the same document, which greatly increased the number of different documents loaded. Again, separate 
analyses were done to handle those differences. 

The number of accesses per document per student is relatively uniform across classes, with each document 
being accessed approximately twice by each student. While there are clear variation in access patterns (e.g., the 
root of the web accounts for nearly one-fifth of the accesses, the two classes have different types of pages, 
different students use the web differently), it is comforting to see that students do, in fact, regularly use the 
course materials. 



Course 


Accesses 


Machines 


Docs 


Students 


A/D 


A/D/S 


A/S/Week 


CS152 


1720 


64 


96 


7 


17.9 


2.6 


35 


w/o book 


1225 


63 


70 


7 


17.5 


2.5 


25 


w/o root 


925 


60 


68 


7 


13.6 


1.9 


19 


CS223 


5794 


94 


195 


24 


29.7 


1.23 


34.5 


w/o images 


5378 


94 


144 


24 


37.3 


1.55 


32 


w/o examples 


5027 


94 


83 


24 


60.6 


2.5 


30 


w/o root 


3914 


91 


81 


24 


48.3 


2 


23 



Table 2: Overall web usage in first seven weeks. Restrictions are cumulative. A/D is 
accesses per day. A/D/S is accesses per day per student. A/S/Week is accesses per student 
per week. 

As [Tab. 3] suggests, students look at a wide variety of materials, but most visits are confined to the course root 
(presumably as a starting point for explorations) and outlines (which, in the webs used in this study, contain 
most of the information). 



Course 


Total 


Root 


News 


Syn. 


Outlines 


Assign. 


Examp. 


Book 


Other 


CS152 


1720 


300 


17 


67 


497 


225 


23 


495 


96 




100% 


17% 


1 % 


4% 


29% 


13% 


1% 


29% 


6% 


in class 


747 


170 


5 


34 


287 


100 


5 


109 


37 




100% 


22% 


1% 


5% 


38% 


13% 


1% 


15% 


5% 


CS223 


5794 


1113 


135 


424 


1297 


1215 


353 


0 


851 




100% 


21% 


30% 


8% 


24% 


22% 


6% 


0% 


16% 


in class 


1836 


323 


51 


151 


619 


258 


75 


0 


359 




100% 


18% 


3% 


8% 


34% 


14% 


4% 


0% 


20% 



6 



Table 3: Page access by category in the first seven weeks of each course. Percentages may 
not total 100% due to rounding. 

As one might expect, the types of documents used varies depending on whether students are inside or outside of 
class. In particular, use of class outlines increases significantly during class sessions. Presumably, this is due to 
a number of factors. First, students are less likely to look at "extraneous” material (such as assignments or 
syllabus) during class sessions. In addition, students who use the web during class time are likely to follow 
along in the current outline, or sometimes refer back to previous outlines for further information. 

5. Summary 

For many of the students in these courses, the in-class use of course webs has had an impact on how they learn 
in class, or at least on how they perceive their in-class learning. During each class period, over half the students 
load web pages (in CS223, the sharing of workstations makes presumed usage even higher). Students who use 
the pages in class report that the web helps them learn better in class, and both poor and good note-takers report 
benefits for their note taking. 
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